Papers with LLMs evaluation
ZhuJiu: A Multi-dimensional, Multi-faceted Chinese Benchmark for Large Language Models (2023.emnlp-demo)
Copied to clipboard
Baoli Zhang, Haining Xie, Pengfan Du, Junhao Chen, Pengfei Cao, Yubo Chen, Shengping Liu, Kang Liu, Jun Zhao
| Challenge: | Various types of LLMs have recently been rapidly developing, such as Llama2 and ChatGLM2 . |
| Approach: | They propose a benchmark that comprehensively evaluates LLMs across 7 ability dimensions covering 51 tasks. |
| Outcome: | The proposed benchmarks are comprehensive and systematic, with a high level of accuracy and authority. |
OpenICL: An Open-Source Framework for In-context Learning (2023.acl-demo)
Copied to clipboard
| Challenge: | In-context Learning (ICL) is a new paradigm for large language model evaluation. |
| Approach: | They propose an open-source toolkit for ICL and LLM evaluation. |
| Outcome: | The proposed framework is highly flexible and flexible and can be easily combined with other tools to suit users' needs. |
S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Model (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks fail to evaluate extremely long-context LLMs or analyze their limitations. |
| Approach: | They propose a Synthetic, Scalable, Systematic evaluation suite for LLMs using SQL execution. |
| Outcome: | The proposed evaluation suite is able to scale text length and difficulty across scenarios and provides strong correlations with real-world benchmarks. |
RAP: A Metric for Balancing Repetition and Performance in Open-Source Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models generate repetitive content, leading to incomplete or fragmented responses, which can negatively affect user experience. |
| Approach: | They propose a new evaluation metric that quantifies and integrates repetition penalty into the assessment of model performance, enabling tuning of RPP. |
| Outcome: | The proposed evaluation metric reduces repetition while minimizing performance loss. |